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Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH(S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
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Application Papers 
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Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 

11) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12) 13 Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)I3 All b)D Some * c)D None of: 

1 Certified copies of the priority documents have been received. 
2.D Certified copies of the priority documents have been received in Application No 



3.D Copies of the certified copies of the priority documents have been received in this National Stage 
application from the International Bureau (PCT Rule 17.2(a)). 
* see the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1) M Notice of References Cited (PTO-892) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) 

3) □ Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 

Paper No(s)/Mail Date . 



4) 0 Interview Summary (PTO-4 13) 

Paper No(s)/MaiI Date. . 

5) □ Notice of Informal Patent Application (PTO-152) 

6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 1-04) 



Office Action Summary 



Part of Paper No./Mail Date 16 



Application/Control Number: 09/546,392 Page 2 

Art Unit: 2623 

DETAILED ACTION 
Continued Examination Under 37 CFR LI 14 

1 . A request for continued examination under 37 CFR 1.114, including the fee set forth in 
37 CFR 1 .17(e), was filed in this application after final rejection. Since this application is 
eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1 .17(e) 
has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 
37 CFR 1.114. Applicant's submission filed on January 14, 2004 has been entered. 

Response to Arguments 

2. Applicant's arguments on pages 10 and 11, filed December 5, 2003, with respect to the 
restriction requirement have been fully considered and are persuasive. The restriction of claims 
23-26 has been withdrawn. 

3. Applicant's arguments on page 1 1, last two paragraphs, with respect to the drawings have 
been fully considered and are persuasive. The objection to Figures 6 and 9 has been withdrawn. 

4. Applicant's arguments on page 12, first paragraph, filed December 5, 2003, with respect 
to claims 8-10 and 19-21 have been fully considered and are persuasive. The 35 U.S.C. 112, 2 nd 
paragraph rejection of claims 8-10 and 19-21 has been withdrawn. 

5. Applicant's arguments on page 13, first paragraph, with respect to claims 1 and 12 have 
been fully considered but they are not persuasive. On page 13, line 3-5, applicant states, "neither 
reference [Suzuki or Kelley] discusses or suggests using predetermined workpiece-robot 
arrangement information (reference models) to orient the robot for the purpose of acquiring or 
preparing to acquire the workpiece." In response to applicant's argument that the references fail 
to show certain features of applicant's invention, it is noted that the features upon which 
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applicant relies (i.e., "using predetermined workpiece-robot arrangement information to orient 
the robot") are not recited in the rejected claim(s). Although the claims are interpreted in light of 
the specification, limitations from the specification are not read into the claims. See In re Van 
Geuns, 988 F.2d 1 181, 26 USPQ2d 1057 (Fed. Cir. 1993). 

6. Applicant's arguments on page 14, third paragraph, with respect to the combination of 
Kelley with Suzuki have been fully considered but they are not persuasive. The teaching in 
Kelley to use reference model matching to acquire a workpiece from randomly arranged 
workpieces is not based on the fact that Suzuki's disclosed method is based on workpieces laid 
flat and arriving one at a time on a conveyor belt. In response to applicant's argument that the 
"rejection's proposed motive to combine Kelley with Suzuki is incorrect," the test for 
obviousness is not whether the features of a secondary reference may be bodily incorporated into 
the structure of the primary reference; nor is it that the claimed invention must be expressly 
suggested in any one or all of the references. Rather, the test is what the combined teachings of 
the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 
F.2d 413, 208 USPQ 871 (CCPA 1981). In response to applicant's arguments against the 
references individually, one cannot show nonobviousness by attacking references individually 
where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 
208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 
1986). 

7. In light of the above responses to applicant's arguments, applicant's arguments with 
respect to claims 1 and 12 have been considered but are moot in view of the new ground(s) of 
rejection. 
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Claim Rejections - 35 USC § 112 

8. The following is a quotation of the second paragraph of 35 U.S.C. 1 12: 

The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the 
subject matter which the applicant regards as his invention. 

9. Claim 26 is rejected under 35 U.S.C. 1 12, second paragraph, as being indefinite for 
failing to particularly point out and distinctly claim the subject matter which applicant regards as 
the invention. 

10. Claim 26 recites the limitation "the arm" in line 2. There is insufficient antecedent basis 
for this limitation in the claim. 

Claim Rejections - 35 USC §103 

11. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 

obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

12. Claims 1-4, 12-15, and 23-25 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Iida et al. (USPN 5,446,835, newly cited, "Iida") in view of Fallon (USPN 4,985,846, 
newly cited). 

13. Regarding claim 1, Iida discloses a robot system having an image processing function for 
determining orientation, or orientation and position of a robot operation on an object of detection 
among a plurality of objects (column 3, line 50-55), the system comprising: a robot (30); a first 
image capturing device (camera 10); a memory (23) storing reference (collative) models, each 
created based on an image of a reference object (work W) captured by said image capturing 
device in a different direction (basic positions, Figure 4), and for each reference model storing 
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information of the capturing direction of its associated image and information of an orientation 
of the robot (hand information) with respect to the reference object, said reference object (work) 
being one of the plurality of objects or having a shape substantially identical to one of the 
plurality of objects (Figure 2; column 4, line 28-40); and a processor (CPU 21) to perform 
matching (collation) processing on the reference (collative) model images (column 4, line 1-16) 
and a working image (video signal) of the plurality of objects (captured by said first image 
capturing device) to select an image of one of the reference models that matches the object of 
detection in the working image (column 4, line 41-59), and to determine the orientation, or the 
orientation and position (directional central coordinate position with a direction) of the robot 
operation to be performed on the object of detection (column 5, line 20-28), the determining 
based on the selected image of the reference object, based on said one reference (collative) 
model and the information of its associated capturing direction, and based on the information of 
the orientation of the robot with respect to the reference object that is associated with said one 
reference model (column 5, line 50-57; column 6, line 22-30). 

14. Iida discloses that reference (collative) models are set in advance on the basis of the data 
specifying a configuration when a plurality of specific portions of a work W take basic positions 
(column 4, line 28-34), but does not disclose that each reference model is comprised of an image 
of a reference object. Fallon discloses an acoustical/optical bin picking system wherein a 
controller is taught the particular shape of a part and the location and identification of a 
predetermined feature on the part by recording an image of the part in digital memory (column 4, 
line 35-42). It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to store a reference model, each comprising an image of a reference object 
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as taught by Fallon in order to compare a video image of a tentatively identified part to the stored 
image in order to locate a predetermined feature on the part used to generate a bearing to identify 
and locate the part to be picked up (column 2, line 4-18). 

15. Regarding claim 12, a robot system having an image processing function for determining 
orientation, or orientation and position of a robot operation on an object of detection among a 
plurality of objects of plural kinds (column 6, line 51-55), the system comprising: a robot (30); a 
first image capturing device (camera 10); a memory (23) storing reference (collative) models, 
each comprising images (see above discussion of claim 1 with respect to Fallon) of each of 
different kinds of reference objects (column 6, line 51-64; column 10, line 21-31) captured by 
said image capturing device, and storing indicia (marks) of the kinds respectively associated with 
said reference models (column 15, line 9-28, line 42-47), and information of a different 
orientation (hand information) of the robot with respect to each of the different images of the 
reference object of each kind, each of said reference objects being one of the kinds of the 
plurality of objects or having a shape substantially identical thereto (Figure 2; column 4, line 28- 
40); and a processor (CPU 21) to perform matching (collation) processing on the reference 
(collative) model images (column 4, line 1-16) and a working image (video signal) of the 
plurality of objects of plural kinds (captured by said first image capturing device) to select an 
image of one of said kinds of the reference models that matches the object of detection in the 
working image (column 4, line 41-59), and to determine the orientation, or the orientation and 
position (directional central coordinate position with a direction) of the robot operation to be 
performed on the object of detection (column 5, line 20-28), the determining based on the 
selected image based on said one reference (collative) model, based on the indicia (marks) of the 
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kind associated with said one reference model and the information of the orientation of the robot 
with respect to the reference object associated with said one reference model of said one kind 
(column 5, line 50-57; column 6, line 22-30). 

16. Regarding claims 2 and 13, Fallon discloses that said reference models are obtained from 
a part of the image data of the reference object (column 4, line 37-42). 

17. Regarding claims 3 and 14, Fallon discloses that said reference models are obtained by 
processing the image data of the reference object (column 4, line 37-42). 

18. Regarding claims 4 and 15, Iida discloses that said first image capturing device comprises 
a camera (10) for capturing two-dimensional image data (column 3, line 50-55). 

19. Regarding claim 23, Iida discloses a method for automatically determining an 
arrangement of a workpiece relative to a robot (column 3, line 50-55), the method comprising: 
storing reference (collative model) images (see above discussion of claim 1 with respect to 
Fallon) of the workpiece or an object so shaped (workpiece/object) and reference arrangement 
information indicating arrangements (basic positions) of the robot and workpiece/object relative 
to each other when the images were captured (column 4, line 28-40); from a known arrangement 
of the robot, capturing a working image of the workpiece among a plurality of randomly 
arranged workpieces with an imaging device (column 4, line 41-46); finding one of the reference 
images that has a closest match (collation) to the workpiece in the working image (column 5, line 
6-22); and determining a known arrangement of the robot relative to the workpiece (directional 
central coordinate position with a direction) based on information indicating the known 
arrangement of the robot, and based on the reference (collative model) arrangement information 
corresponding to the found reference image (column 5, line 20-28; column 6, line 22-30). 
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20. Regarding claim 24, Iida discloses that reference images and reference arrangement 
information is obtained for workpieces/objects of different shapes (column 6, line 51-55), and 
wherein the finding comprises first determining that a reference image of one of the different 
shapes matches the working image of the workpiece, and then finding one reference image of the 
shape that best matches the working image (column 6, line 55-64; column 10, line 21-31). 

2 1 . Regarding claim 25, Fallon discloses that the robot is used to capture the reference 
images, and wherein the reference arrangement information represents arrangements of the robot 
when capturing the reference images (column 4, line 35-42). 

22. Claims 5 and 16 are rejected under 35 U.S.C. 103(a) as being unpatentable over Iida in 
view of Fallon as applied to claims 4 and 15 above, and further in view of Suyama et al. (USPN 
4,879,664, previously cited, "Suyama") or StaufFer (USPN 4,410,804, previously cited). 

23. Iida does not disclose that the image data are captured from a predetermined distance. 
Suyama discloses a three-dimensional position sensor comprising robot-teaching apparatus 
wherein said image data of the reference object are captured by said camera (figure 1 la, element 
35) from a predetermined distance (column 6, line 20-37). In addition, Stauffer teaches that if a 
two-dimensional sensor is used, the image processor is unable to simultaneously determine the 
distance to the object unless the objects are always positioned at a known distance (column 1, 
line 3 1-39). It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to capture an image at a predetermined distance in order to calibrate a three- 
dimensional sensor as taught by Suyama or such that the distance to the object need not be 
determined by other means as taught by Stauffer. 
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24. Claims 6, 7, 11, 17, 18, and 22 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Iida in view of Fallon as applied to claims 1 and 12 above, and further in view of Maeno et 
al. (USPN 5,047,714, previously cited, "Maeno"). 

25. Regarding claims 6 and 17, Iida does not disclose that the robot moves a second image 
capture device to have the determined orientation and/or position. Maeno discloses a method of 
recognizing surface-mounted parts including a robot system comprising a second image 
capturing device (figure 7c, element 10); wherein said robot situates said second image data 
capturing device to have said determined orientation or to have said determined orientation and 
said determined position with respect to the object (column 4, line 47-49), and said processor 
processes second image data captured by said second image capturing device to detect position 
and/or posture of the object with respect to said second image data capturing device (column 4, 
line 54-63). It would have been obvious to one of ordinary skill in the art at the time the 
invention was made to move the second image capture device (imaging tube) to have the 
determined orientation and/or position as taught by Maeno in order to locate position detecting 
patterns located at specific positions to accurately and directly determine the position and 
direction of conductor patterns on the printed circuit board (column 1, line 51-59). 

26. Regarding claims 7 and 18, Iida discloses determining the three-dimensional position 
(directional central coordinate position with a direction) of the specific portion of the work 
searched by the collative model, but does not disclose that the robot moves a second image 
capture device to have the determined orientation and/or position. Maeno discloses that said 
robot situates said second image data capturing device to have said determined orientation or to 
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have said determined orientation and said determined position with respect to the object, so that 
the second image data capturing device is directed to a characterizing portion of the object; and 
wherein said processor detects the position and/or posture of the object based on the position of 
said characterizing portion obtained by said second image data capturing device (see discussion 
of claims 6 and 17 above). 

27. Regarding claims 1 1 and 22, Iida discloses that said robot operation is an operation of 
picking up at least one object from a plurality of objects overlapped with each other (column 3, 
line 44-55). 

28. Claims 8, 19, 27, and 30 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Iida in view of Fallon in view of Maeno as applied to claims 6, 7, 17, and 18 above, and further 
in view of Soderberg (USPN 4,785,528, newly cited). 

29. Regarding claims 8, 19, 27, and 30, Maeno does not disclose that said first image data 
capturing device is used as said second image data capturing device. Soderberg discloses a 
robotic work positioning system including a camera 34 mounted on member 28 of robot 10 
which serves as both the first and second image data capturing device since it both determines 
the position of the work, and is moved to have the determined orientation and/or position (Figure 
1; column 2, line 53-column 3, line 1, line 44-50; column 5, line 34-41). It would have been 
obvious to one of ordinary skill in the art at the time the invention was made to use said first 
image data capturing device as said second image data capturing device as taught by Soderberg 
in order to retrieve parts from a conveyor using a vision system to select the desired part from 
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other parts and engage the part at precisely the correct location to be properly placed by the robot 
(column 5, line 60-column 6, line 6). 

30. Claim 26 is rejected under 35 U.S.C. 103(a) as being unpatentable over Iida in view of 
Fallon as applied to claim 23 above, and further in view of Soderberg. 

3 1 . Regarding claim 26, Iida does not disclose that a second imaging device is affixed to the 
arm (of the robot; see above 35 U.S.C. 1 12, 2 nd paragraph rejection). Soderberg discloses a 
camera 34 mounted on member 28 of robot 10 and is used to determine additional arrangement 
information used to determine the known arrangement of the robot relative to the workpiece 
(Figure 1; column 3, line 44-50; column 5, line 34-41). It would have been obvious to one of 
ordinary skill in the art at the time the invention was made to affix an imaging device to the arm 
of a robot as taught by Soderberg in order to retrieve parts from a conveyor using a vision system 
to select the desired part from other parts and engage the part at precisely the correct location to 
be properly placed by the robot (column 5, line 60-column 6, line 6). 

32. Claims 9, 10, 20, 21, 28, 29, 31, and 32 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Iida in view of Fallon in view of Maeno as applied to claims 6, 7, 17, and 18 
above, and further in view of Sakakibara et al. (JP 07-270137, previously cited, "Sakakibara"). 

33. Regarding claims 9, 20, 28, and 3 1, Maeno does not disclose that said second image 
capturing device comprises a three dimensional visual sensor of spot-light scanning type. 
Sakakibara discloses a three dimensional visual sensor usable in robot automation (page 1, 
paragraph 1), of spot-light scanning type capable of measuring distance between the sensor and 
an object (page 3, paragraph 15). It would have been obvious to one of ordinary skill in the art at 
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the time the invention was made to utilize a three dimensional visual sensor of spot-light 
scanning type as taught by Sakakibara in order to quickly and accurately determine the three 
dimensional position of an object using one device (page 2, paragraph 1 1). 

34. Regarding claims 10, 21, 29, and 32, Sakakibara discloses an image data capturing device 
comprising a structured-light unit for irradiating a structured light on an object and capturing an 
image of the object including the irradiated light on the object (page 3, paragraph 16). 

Conclusion 

35. The prior art made of record and not relied upon is considered pertinent to applicant's 
disclosure. 

• USPN 4,462,046 to Spight discloses a machine vision system producing a video 
signal representative of a spatial orientation and position of a viewed object. 

• USPN 4,680,802 to Nishida et al. discloses a posture judgment system in image 
processing at the time of the preparation of standard data. 

• USPN 4,909,376 to Herndon et al. discloses a robotically controlled component feed 
mechanism visually monitoring part orientation. 

• USPN 5,727,132 to Arimatsu et al. discloses a robot controlling method for tracking a 
moving object using a visual sensor. 

• USPN 6,349,245 to Finlay discloses a method and apparatus for registration of a 
robot carrying an image acquiring arrangement. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Ryan J Hesseltine whose telephone number is 703-306-4069. 
The examiner can normally be reached on Monday - Friday, 8:30 AM - 5 PM. 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's / 
supervisor, Amelia Au can be reached on 703-308-6604. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 



rjh 

March 18, 2004 



